A Critique of E-Commerce Website Evaluation Models

Quality has been established as a key factor in ensuring the success of E-commerce in attracting and retaining customers. To help in this, numerous software metrics and website quality models have been developed, with a correspondingly large literature. We provide a critical review of this literature and the state-of-the-art. Most of the wide ranges of E-commerce website evaluation models give emphasis on the web applications of the system, using techniques like feature inspection and collecting data about end-users’ opinion by questionnaires. However, this is in conflict with two fundamental pragmatic aspects of current websites. Web technologies evolve extremely fast, enabling sophisticated tools to be deployed and complex interactions to take place. Secondly, the life cycle of a website is also extremely fast: maintenance of a website is performed at a rate that is higher than that of other software products because of market pressure and lack of distribution barriers. Over 45 scholarly models of website quality have appeared in the last 10 years. A small sample of those studies had been tested on over 436,000 data points from 16,000 respondents. What this indicates is that the application and use of scholarly models of website quality is a very-well established discipline. However, many of these models have numerous factors and sub-factors, as well as unusually large measurement instruments that demands extra time for data collection and data analysis in each measurement phase, which are economically prohibitive to apply. Also, many of these models have not proven very robust, and exhibit low levels of reliability and validity. In this paper, we recommend a holistic model for E-commerce website evaluation, using Bayesian Belief Networks, as alternative approaches to the single-issue models used at present. This model differs from questionnaire-based surveys approaches in that it uses a process aiming to limit subjectivity and frequent errors in similar surveys and provides a flexible way to define the quality of E-commerce websites, as users perceive it, in a short period of time. 1. OVERVIEW OF WEBSITE EVALUATION The common issues found in the literature relating to website evaluation are quality (e.g. Dran et al., 1999, Cox and Dale, 2002), Web design (e.g. Gehrke and Turban, 1999, Thelwall, 2003), and usability (e.g. Nielsen, 1995, Konradt et al., 2003). Researchers have adopted the Web quality concept from the quality of product or service (e.g. Cox and Dale, 2002). For example, Dran, Zhang, and Small (1999) adopted Kano’s Model of Quality as a theoretical framework to evaluate the quality of websites. This model separated product and service quality into three levels according to customer expectations: expected, normal, and exciting. These researchers believe that quality in a product or service is not what the provider or seller put into it, but what the client or customer receives from it. In regard to Web design, Shneiderman (1997) provided an Objects/Actions Interface (OAI) model for Website design. This encourages designers of a website to focus on analyzing the relationship between the task and Web interface. Wan and Chung (1998) looked at problems in Web design from the perspective of network analysis. They suggested that care must be taken when designing the homepage, which is the entrance to the website. Gehrke and Turban (1999) suggested five major categories that should be considered when designing a website for a business: page loading, business content, navigation efficiency, and security and marketing/consumer focus. They argued that page loading is the most important factor in website design. Undertaking a usability study usually needs high consumer or user involvement, and sometimes the study needs to be conducted in an experimental environment. Nielsen (1995) provided guidelines and criteria to evaluate the usability of website design and suggested that every design project, including website development, should be subjected to usability testing and other validation methods. Toh and Pendse (1997) also suggested that Web pages should be designed for usability and understanding. However, a website with good usability cannot guarantee users’ preference. 2. MEASURING AND ANALYZING E-COMMERCE QUALITY The measurement of quality in information technologies has been an issue of concern for a long period of time. This issue has had a great deal of attention from many researchers in the academic world (Seddon et al, 1999; and Delone and Mclean, 1997). Lehman and Belady (1985) established a simple classification for information systems, being either E-type or S-type. An S-type system is one that is completely and totally defined, and is required to be correct with respect to a mathematically defined specification. An E-type system, on the other hand, resolves to expectations of the system. An E-type system is correct when it satisfies the user expectations. A classification of information systems in terms of its quality indicators, categorized quality to three perspectives: product, process and service (Whyte and Bytheway, 1996). Various studies related to the three perspectives have produced a number of measures for evaluating informational systems such as E-commerce websites. These include system usage (Srinivasan, 1985), information value (Gallagher, 1974) and user satisfaction (Bailey and Pearson, 1983). The diversity of these various measures was initially a cause for concern, so Delone and McLean (1997) attempted to synthesize them into a unified model. The Delone and McLean (1997) Model of “Information Services Success” has been regarded by many authors as a major contribution (Molla and Licker, 2001) and has been the focus of several studies (e.g. Seddon et al., 1999). Pitt and Watson (1997) proposed a modification of this model to include a “Service Quality” component. This modification was endorsed by Delone (2003) together with other modifications integrated to the updated Information Services Success Model (Delone, 2003). Some researchers have highlighted the problem of inadequate measures for assessing the benefits of investments in Information Technology (Molla and Licker, 2001). There is a considerable difficulty in measuring the quality of informational systems and there lies some difficulty in searching for appropriate metrics. Notwithstanding the literature review concerning the difficulty in developing measures, there is still a need for an indicator of the success of a company’s E-commerce website. One possible indicator is that of user satisfaction. Various sources have argued that measuring satisfaction of users is useful as a surrogate indicator of information system quality. The utilization of user satisfaction for measuring quality is discussed in the next section. 3. USER SATISFACTION AS AN EFFECTIVE MEASURE User satisfaction gradually became a measure of software quality during the 1950s, 1960s, and 1970s (Thayer, 1958; Hardin, 1960; Lucas, 1974). User satisfaction is defined as “the sum of one’s feelings or attitudes toward a variety of factors affecting that situation,” e.g., computer use and adoption by end users (Bailey and Pearson, 1983). Most studies until 1980 focused on the end user’s satisfaction toward software developers; but one study squarely focused on the end user’s satisfaction with the software itself (Lyons, 1980). Pearson and Bailey (1980) produced one of the first studies to address a variety of software attributes such as software accuracy, timeliness, precision, reliability, currency, and flexibility. Studies throughout the 1980s addressed user satisfaction with both designers and software (Walsh, 1982; Bailey and Pearson, 1983). The late 1980s marked a turning point with studies focusing entirely on user satisfaction with the software itself and attributes such as content, ease of use, and timeliness of the software (Doll and Torkzadeh, 1988). A study of user satisfaction at IBM was based on reliability, capability, usability, installability, maintainability, performance, and documentation factors. Throughout the 1990s, IBM used a family of user satisfaction models called UPRIMD, UPRIMDA, CUPRIMDA, and CUPRIMDSO, which referred variously to factors of capability, usability, performance, reliability, installability, maintainability, documentation, availability, service, and overall satisfaction (Kan, 1995). User satisfaction, now commonly referred to as customer satisfaction, is no doubt related to earlier measures of software attributes, usability or user friendliness of software, and more recently, web quality. In E-commerce, interaction with the end-user is conducted through webbased applications including both server and client-side applications commonly referred to as a website. All usersystem communication is realized through the interface, so it is self evident that the quality of an E-commerce system is directly related to the quality of the user interaction experience (Ζwass, 1996). Research efforts by Wang (2003) have directly tied the assessment of an E-commerce website to customer satisfaction. A survey carried out by Wang (2003) on 35 E-commerce companies in the United States identified three proponent methods for assessing quality. All three were actually an assessment of the satisfaction of the customer. The three major assessment methods are text comments, categorized rating and overall rating. Text comment allows customers to write their own comments in 500 to 1000 characters on the “where”s and “why”s they did their shopping. Categorized rating is achieved with a questionnaire that asks online shoppers to rate a number of quality determinants using a scale of 1 to N where N is the best rating. The overall satisfaction rating uses an ordinal rating system with a scale of 1 to N where N is the best rating. User satisfaction is a combination of experience and perception (Gefan, 2002). It has been shown that several factors can positively or negative influence a user’s experience and their perception of a website experience (Stefani et al., 2003). 4. EVALUATING E-COMMERCE WEBSITES: A REVIEW OF EVALUATION CRITERIA Website quality models appearing in the late 1990s, following the user satisfaction movement appeared as important measures of software quality (Lindroos, 1997). One of the first models of website quality identified background, image size, sound file display, and celebrity endorsement as important factors of software quality (Dreze and Zufryden, 1997). The web assessment method or WAM quickly followed with quality factors of external bundling, generic services, customer specific services, and emotional experience (Selz and Schubert, 1997). In what promised to be the most prominent web quality model, attitude toward the site had quality factors of, informativeness, and entertainment (Chen and Wells, 1999). The next major model was the e-satisfaction model with its five factors of convenience, product offerings, product information, website design, and financial security. The website quality model or WebQual for business school portals was based on factors of ease-of-use, experience, information, and communication and integration. An adaptation of the service quality or ServQual model, WebQual 2.0 measured quality factors such as tangibles, reliability, responsiveness, assurance, and empathy (Barnes and Vidgen, 2001). Although some researchers have tried to provide ways of evaluating E-commerce website specifically (e.g. van der Merwe and Bekker, 2003), the selection of evaluation criteria still requires more theoretical justification. A selection of evaluation criteria is shown in Table 1; each of these has their points of strengths and weaknesses. Studies on E-commerce website quality also focus on more specific quality characteristics such as issues that warrant successful transactions (Bidgoli, 2002), maximize the perceived trustworthiness (Egger, 1998), or ensure E-commerce website reliability (Dustin et al., 2001). Although, all the above factors affect the quality of E-commerce websites and are prerequisites for their success, they are not the only ones that relate to E-commerce website quality. Farthing and Stocking (2005) jumped to a conclusion that there is no fully integrated approach after their review of the literature. From these previous studies, it can be inferred that a global approach, such as the one discussed in this paper, is required combining all factors affecting quality. 5. A CRITIQUE OF CURRENT APPROACHES TO EVALUATING E-COMMERCE WEBSITE Early definitions of software quality included fitness for use, conformance to requirements, or degree to which software satisfied its specified requirements. These classical definitions of software quality imply one must gather customer requirements, develop a software product, and then determine how many quality requirements have been satisfied. Since the 1960s, increasingly sophisticated views of software quality have emerged: software size, software errors, software attributes, software defect models, software complexity, software reliability, user satisfaction, and website quality, to name a few. One of the earliest approaches for measuring software quality was the practice of quantifying and assessing attributes or characteristics of computer programs. Software attributes are traits, characteristics, features, or other properties of software products. Early studies attempted to enumerate, qualify, and quantify all of the attributes of software products. One such study (Boehm et al., 1978) identified the following attributes: correctness, efficiency, flexibility, integrity, interoperability, maintainability, portability, reliability, reusability, testability, and usability. Throughout the 1970s and 1980s the practice of measuring software attributes waned in favor of statistical models of software quality and reliability, which estimated defects and mean time to failure. However, during the 1990s, the practice of measuring software attributes began to take a foothold once again in the form of user satisfaction and website quality models. User satisfaction models were used to measure end user attitudes towards software products. One such model (Barnes and Vidgen, 2002) measured user attitudes about the following attributes of software quality: usability, design, information, trust, and empathy. Models of user satisfaction were eventually overtaken by models of website quality by the end of the 1990s. Basic website quality is defined as a “customer’s judgment about the website’s overall excellence or superiority, which is an attitude that comes from a comparison of expectations and perceived performance”. Within the context of Ecommerce, website quality refers to “the extent to which a website facilitates efficient and effective shopping, purchasing, and delivery of products and services”. According to Frico (2007), over 45 scholarly models of website quality have appeared in the last 10 years. Table 1A review of evaluation criteria Reference Perspective Strengths Weaknesses Kramer, 2000 Evaluating E-commerce servers Evaluating E-commerce servers Practical advice for system managers Evaluates E-commerce server technology not web sites or customer service (Barnes & Vidgen, 2002) WebQual HCI framework: Five factors identified: Usability, design, information, trust & empathy Based on customer perceptions of quality weighted by importance Very narrow focus (Schubert & Dettling, 2002) EWAM EWAM (Extended web assessment method ) is a tool specifically created for the evaluation of E-commerce sites Considers sites from the customers’ perception Concentrates on generic web issues with little consideration of issues important to selling Akhter, et al., 2005 Evaluating consumer trust Compares customers’ trust with their familiarity with the site, and objective measures of security. Identifies how important these are in encouraging trust Intended solely to evaluate trust, not other factors important to selling Lightener, 2004 Evaluation from a customer service perspective Consideration of the design is subordinate to the functions actually provided Doesn’t cover the whole selling life cycle, e.g. customer finding the site, generating repeat business Hahn et al., 2002 Evaluation from an investment perspective Management focus Identifies problem areas but not solutions A small sample of those studies had been tested on over 436,000 data points from 16,000 respondents (Frico, 2007). What this indicates is that the application and use of scholarly models of website quality is a very-well established discipline. However, many of these models have numerous factors and sub-factors, as well as unusually large measurement instruments, which are economically prohibitive to apply. Also, many of these models have not proven very robust, and exhibit low levels of reliability and validity (Frico, 2007). Most of the tools that have been developed for the assessment of E-commerce websites give emphasis on the web applications of the system and they are based on surveys (Molla and Licker, 2001). This process provides significant results but demands extra time for data collection and data analysis in each measurement phase. The work presented in this paper, differs from questionnaire-based surveys in that it uses a process aiming to limit subjectivity and frequent errors in similar surveys and provides a flexible way to define the quality of E-commerce websites, as users perceive it, in a short period of time. 6. PREDICTING E-COMMERCE QUALITY Given that the establishment of an E-commerce website is mainly a software development effort; there are several standards that apply in governing the quality of such development. According to de Chazal (2005), there seems to be an almost overwhelming abundance of quality standards that lead to a high level of cynicism and skepticism surrounding them and the eventual lack of use. Website developers need to use standards and best practices to ensure that websites are functional, accessible and interoperable. However many websites fail to achieve such goals and no standard can directly predict the quality a website under development is going to achieve. The software behind any E-commerce website is, in essence, the virtual organization and business operation of that site. It is thus reasonable to conclude that the quality and evaluation methods of E-commerce systems will always be dependant on the quality of applications they contain and their ability to meet end-user requirements. An E-commerce website can be assessed by the quality factors of its software. Having these quality factors enables the measured specification of attributes and variables. Such quality factors should be seriously considered during the development of E-commerce websites (Stefani et al., 2003). Past approaches concerning the quality of E-commerce websites emphasized the usability standards, using techniques like feature inspection methods and collecting data about end-users’ opinion by questionnaires. These methods provide an important feedback and their results are of useful background for future work, however, they do not contribute directly to a dynamic model that enables forecasting (Chan et al., 2001). In this paper, a model is proposed where the attributes are of a dynamic character. The results derived from the application of the proposed model are utilized to predict E-commerce website quality and to direct the development of a website to increase the quality measures, producing a site that gives an E-commerce experience with high service quality and user satisfaction. Furthermore, the results derived from its application are utilized for the model’s constant improvement, thus contributing to a continuous evolvement and upgrading. 7. MOTIVATION FOR APPLYING BAYESIAN BELIEF NETWORKS APPROACH Having a metric for quality makes matters easier for a business, as it can then measure whether quality is being attained. Seddon et al (1999) define quality as “a relative value that is meaningful only when compared to postulated values that are defined by the user or by standards organizations.” Several researchers such as McCall et al. (1977) and Boehm et al. (1978) have since proposed holistic quality models incorporating a wide array of measures, in order to define a quality system. According to de Chazal (2005), holistic models such as these often require substantial infrastructure in order to capture and analyze the data gathered. Consequently, many companies look for easier alternatives, such as a single measure of quality, as opposed to process-driven quality. Niedermayer (1998) describes a Bayesian Belief Network (BBN) as a model that defines various events, the dependencies between them, and the conditional probabilities involved in those dependencies. The mathematical model on which Bayesian Belief Networks are based is the theorem developed by the mathematician and theologian, Thomas Bayes. The BBN is a special category of graphic models where nodes represent variables and the directed arrows represent the relations between them. Therefore, a BBN is a graphical network that describes the relations of probabilities between the variables (Agena, 2006). This information can then be used to calculate the probabilities of various possible causes being the actual cause of an event. A framework for assessing the qualities of an E-commerce website is the essence of this paper. Now, the question which arises is: ‘Can a Bayesian Belief Network be applied to anticipate the level of quality of the site and the factors behind that level of quality?’ According to Advocate.com (Advocate, 2001), in applying a Bayesian Belief Network, a single model can be used for both diagnostic and causal reasoning. That is, the same model can be used to reason from effects to causes and from causes to effects. This suggests that a Bayesian Belief Network could be used to systematically predict the qualities of an E-commerce website under development and to determine the reasons for the predicted quality. 8. A PROTOTYPE BBN MODEL FOR E-COMMERCE WEBSITE While there is insufficient space here to fully describe the development and execution of a BBN model here we have developed a prototype BBN to show the potential of BBNs and illustrate their useful properties. With this model, we should be able to show how assessments might be made. The philosophy underlying the BBN model is the creation of a dynamic network that concentrates and exploits the knowledge gained from the analysis of data gathered during previous researches and that can also use its own results for future estimations. A graphical presentation of the network is illustrated in Figure 1. Figure 1: Execution of the model using Hugin Expert A/S tool – The model uses nodes to represent the quality factors, characteristics and sub-characteristics of E-commerce websites. Each node is characterized by a set of possible states called evidence and is connected to its parent nodes by directed arrows. In figure 1 the node ‘Quality’ represents the E-commerce website quality as a whole and is characterized by three possible states (evidence): ‘Yes, ‘Perhaps’, and ‘No’. The parent nodes of ‘Quality’ are the nodes: ‘Conceptual Reliability, ‘Usability’, and ‘Representative Reliability’. These quality factors characterized by three possible states: ‘positive, ‘Neutral, and ‘Negative. Each quality factor node is connected to the corresponding E-commerce websites quality characteristics, based on our previous research (Rababah et al., 2006a and Rababah et al., 2006b). Finally, each of these quality characteristics is connected to a number of child nodes comprising the quality sub-characteristics of E-commerce Websites. The model has been developed using Hugin Expert A/S. An example of the tool’s user interface is shown in Figure 1. Each node of the model has a Node Probability Table that presents the discrete conditional probability distribution. This table presents the relations between this node (child node) and its parent nodes. One of the most important factors affecting the successful application of the model is the definition of the Node Probability Table of each node. If the probabilities are based on accurate data that have been systematically collected, the estimation will be accurate. However, even in the case that the data of the Node Probability Tables are not completely accurate, the model can still provide results. It can learn (collect experience) and improve the results it provides. The model can be used both forwards and backwards. Backward use of the model provides assessments regarding the child nodes (e.g. nodes of E-commerce characteristics) when the value of a parent node (e.g. node of ‘Conceptual Reliability’ characteristic) is defined. In the forward use, the values of each node are the inputs that are gathered from the evaluators’ answers. These answers have only three possible states: ‘positive’, ‘negative’ and ‘neutral’. In this way, the model estimates the website’s quality providing the probabilities for the possible states of the nodes that represent the quality characteristics and the overall quality of the website. Figure 1 shows the execution of the model using the Hugin Expert A/S tool. Each of the nodes is shown as a window with a histogram of the predictions made based on the facts entered (facts are represented by histogram bars with 100 percent probability).